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WINDOWS API TRAPPING SYSTEM 



Cross-Reference to Related Applications 

This application is based on U.S. provisional patent application No. 60/028,339, 
filed on October II, 1996. 

Background of The Invention 

1 . Field of the Invention 

This application relates to the field of software and more particulariy to the field of 
mana^ng aspects of an interface between software and the underlying operating system. 

2. Description of Related Art 

Many conventional operating systems provide a formalized Application 
Programming Interface (API) that allows application programmers to make calls to 
software routines that perform a variety of system-wide functions. Using the API 
fiidlitates writing application programs by decreasing the amount of code that application 
programmers need to provide and, at the same time, providing standardization of routines 
that are used by many of the applications. 

However, in some instances, it is necessary to modify an API call in order to, for 
example, perform specialized fiinctions that are not provided by the operating system or to 
keep track of certain types of API calls. For some operating systems, it is not difficult to 
intercept and monitor API calls. For example, under MS-DOS, most APIs use interrupts 
and are thus easy to intercept. In other instances, intercepting and trapping API calls can 
be challen^ng for an application programmer. For example, in some Microsoft Windows 
Environments (Winl6 and Win32), the APIs use exported functions which are connected 
to the application at application load time via static or dynamic linkirfg. This linkage 
process is done by the internal OS routines and undocumented data structures vdiich are 
usually inaccessible to outside (non-Microsoft) software developers. In addition, newer 
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veraons of the Windows OS activdy thwart API intercepting methods described in the 
programming literature for the earlier versions of the OS. In some instances, a 
technolo^cal race is unfolding between the operating system developers and the third 
party (application) developers who need to provide OS/applications enhancements 
5 unforseen (or undesired) by the OS developers, 

A method for intercepting Windows 3.x APIs based on patching the entry point of 
the API function with a JMP instruction is known. In this system, the intercqitor first 
obtains the address of the target API function via a call to GetProcAddressQ. Note that 
this step has been actively thwarted by the OS in Win95 for many key API functions. 

10 although within months the workarounds for much of the thwarting attempts have been 

published. Following getting the procedure address, the interceptor removes the 
write-protection of the obtained memory address (which is a code address, thus it is set by 
the OS as read/execute-only). Following this, the interceptor patches the API entry point 
with JMP InterceptSrv instruction, where InterceptSrv is a service function in the 

15 interceptor's code that monitors and/or processes the API calls. 

After the interceptor has set up the API call in the manner described above, an 
application or OS calls the API entry point. The JMP InterceptSrv instruction at the 
entry point address transfers control to the InterceptSrvQ function and the InterceptSrvQ 
processes the call (accessing as necessary the function arguments on the stack) and, upon 

20 completion, either returns control to the caller or passes control to the original API 

function. Returning control to the caller is straightforward. Passing control to the API, 
on the other hand, requires that the InterceptSrvQ remove the patched in JMP 
InterceptSrv from the API entry point and restore the ori^nal opcode bytes (the original 
five bytes that were present in the API before the patch was inserted). Then, the 

25 InterceptSrvQ pushes all original arguments fi-om the-caller's stack on to the current stack 

(interceptor's stack). Finally, the InterceptSrvQ calls the original API function. When the 
original API returns, the InterceptSrvQ saves the return value into a local variable, the 
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IntereeptSrvO reinserts the JMP InterceptSrv patch at the API entry point, and the 
InterceptSrvQ returns the saved return value to the originaJ caller. 

The technique described above is used in many commercial application enhancers. 
However, it has many drawbacks. For one thing, the division of labor between the set up 
5 portion and the run time portion is highly inefficient since the set up portion is executed 
only once (at the interceptor's load time) while the mn time portion is executed many 
times (from the load time onward). In a more efficient system, as much of the work as 
possible would be shifted from the nin time portion to the set up portion. Li addition. fi>r 
a processor ninning MS Windows on an Intel processor, the necessary replacement of the 
10 JMP Intercept opcodes requires a write operation of five bytes, which can not be done in 
a single processor instniction. Hence, the replacement leaves a short interval of instabiUty 
between the two write instructions, during which the API entry point has invalid 
instnictions and a hardware interrupt at that moment could initiate reentry into the API. 
which will probably crash the system. 

°* l^and, disabling the btenupts in Windows appUcation mode (CPU 
ring 3 code where these actions are occurring) is a highly expensive operation due to 
system control over the CPU imerrupt flag, which triggers an elaborate exception process 
Cm CPU ring 0). The complex ring transition and the exception process, which may take 
as much or more time than all the rest of processing above, would occur twice. Therefore 
20 most commercial interceptors (as weU as the published code) choose the tradeoff with the 
instabiUty aUowed. in onler not to pay the disproportionate perfbnnance cost associated 
with disabling interrupts. 

Another disadvantage of the technique described above is that, during certain time 
intervals, another call to the monitored API wiU be missed by the interceptor. Since many 
25 Windows APIs perform checks on the task queue within and can (and often do) switch to 

another thread/task, the missing of intercepts is a real problem for interceptors which 
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require processing on every call to the API (espedally those implementing security 
features). Also, the overwriting of the API entiy point on every call is unsafe when 
multiple intercq>tors exist on the same system. For eTcampIe, while the system is 
processing the original API call, a switch to another task or thread can (and often will) 
5 occur. If the second task inserts its own intercept for the same API (or spawns a program 

which does that), then return to the first interceptor will destroy the new int^cept, thus 
permanently disabling operation of the new intercept. Additionally, if the first intercept 
imloads, followed by an unload of second intercept, then the API entiy will be left pointing 
to the non-existent first intercept and the system will crash v/hm the API is invoked. 

10 Another disadvantage of the technique described above is that some Windows API 

functions (e.g. memory allocation and protection) are sensitive to the source of the call, 
Le. the Windows API code wll check where the call is made fi-om and, based on 
knowledge of Microsoft's sources of calls, will work differently if called from third party 
applications as opposed to particular Microsoft sources. This behavior is, among other 

15 reasons, related to the active thwarting of the third party interceptors mentioned above. 

Since the intercept technique described above changes the original source of the API call 
and makes the interceptor appear to Windows as the source of the API call, the Windows 
API processing may operate differently, often malfimctioning in a way that leads to 
instabilities and system crashes. This makes the technique described above unsuitable for 

20 intercepting some of the Windows APIs. 

Summary Of The Invention 

According to the present invention, supplementing a software routine loaded in 
computer memory includes loading an additional routine into the computer memory, 
providing relocated opcodes by relocating a number of bytes from a relocatable portion of 
25 the software routine to an other memory location where the number of bytes corresponds 

to an integral number of instructions of the relocatable portion, causing program control 
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to flow from the additional routine to the relocated opcodes, causing program control to 
flow from the relocated opcodes to a memory address immediately following the 
relocatable portion, and causing program control to flow from the relocatable portion to 
the additional routine. 

5 Causing program control to flow from the additional routine to the relocated 

opcodes may include placing the relocated opcodes at a location in the computer memoiy 
that immediately follows the additional routine. Causing program control to flow from the 
relocated opcodes to the memory address immediately following the relocatable portion 
may include placing a program control instruction at a location in the computer memoiy 

10 immediately following the relocated opcodes. The program control instruction may be an 

unconditional jump instruction. Causing program control to flow from the relocatable 
portion to the additional routine may include placing a program control instruction at a 
memory location corresponding to a source address of the relocatable portion of the 
software routine. The program control instruction may be an unconditional jump 

15 instruction. Providing the relocated opcodes may include relocating a number of bytes 

that is at least equal to an amount of bytes required for the unconditional jump instruction. 

Followng providing the relocated opcodes, it is possible to resolve any opcodes 
contained therein that reference relative displacements between the relocated opcodes and 
opcodes contained in the software routine. The software routine and additional routine 
20 may be APFs that run under the Microsoft Windows operating system. The additional 

routine may be configured to load at a predetermined address in the computer memory. 

According fiirther to the present invention, supplementing a Windows API loaded 
in computer memory includes loading an additional routine into the computer mwnoiy, 
providing relocated opcodes by relocating a number of bytes from a relocatable portion of 
25 the API to an other memory location where the number of bytes corresponds to an 

integral number of opcodes of the relocatable portion, causing program control to flow 
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from the additional routine to the relocated opcodes, causing program control to flow 
from the relocated opcodes to a memory address immediately following the relocatable 
portion, and causing program control to flow from the relocatable portion to the 
additional routine. According further to the present invention, a software program that 
supplements a Windows API loaded in computer memory includes an additional routine 
that is loaded into the computer memory, first means for providing rdocated opcodes by 
relocating a number of bytes from a relocatable portion of the API to an other memory 
location where the number of bytes corresponds to an integral number of instructions of 
the relocatable portion, second means, coupled to the first means and to the additional 
routine, for causing program control to flow from the additional routine to the relocated 
opcodes, third means, coupled to first means and to the API, for causing program control 
to flow from the relocated opcodes to a memory address inrunediately following the 
relocatable portion, and fifth means, coupled to the API and to the additional routine, for 
causing program control to flow from the relocatable portion to the additional routine. 

In the intercept install phase, the technique described herein disassembles the target 
API entry code and relocates (based on the semantics of the instructions found there) the 
whole instructions from the API entry into the interceptor's memory. Then, in the intercept 
operation phase, when control needs to be passed to the original API fimction, instead of 
having to swap back and forth the overlayed API entry opcodes, the intercept simply 
passes control to the second (relocated) copy of the API entry code, whidi, upon 
completion, passes control to the next section of the original API code (the section which 
foUows the relocated section). 

This method thus shifts the division of labor heavily toward the install phase of the 
intercept, relieving the active phase of the intercept by eliminating many of the steps 
described in connection with the prior art system and the performance, safety and system 
stability drawbacks associated with them. 
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The technique described herdn has many advantages over conventional trapping 
systems. One advantage is that all trapping work, except for a minimum amount of work 
necessary to transfer control to the interceptor and back, is done only once at install time, 
relieving the performance burden from the run time activity of the interceptor In addition, 
no opcode swapping is done during the existence and activation of the traps. This 
eliminates performance, stability and safety drawbacks resulting from the activity found in 
some conventional systems. Furthenmore, since the execution of the some of the trapping 
code occurs on the stack of the original caller, there is no need to copy API function 
arguments to the interceptor's stack. By executing much of the code on the original 
caller's stack, the source of the call will appear to the Windows API as if it came from the 
original caller, therefore resolving problems associated wth Windows behaving differently 
depending on the identity of the caller. By not copying API entry opcodes back and forth 
at each API call, no window of system instability is created. Instead, time-consuming 
precautions (e.g., disabling interrupts) occurs once at intercept install time. 

There are additional advantages. Since the trap opcodes are never removed during 
processing of the original API, the possibility of missing API calls is significantly 
decreased, as described above. Therefore, the system described herein is an excellent 
choice for situations where the interceptor must see all of the API calls to the target API 
function to operate properiy or reliably. The problem of multiple mterceptors setting traps 
while processing of the trap is going on is resolved, since the new interceptor vwll always 
see fixed opcodes at the API entry point, thus the second interceptor wll not be forcibly 
disabled, as in conventional techniques. Also, since the setting and removal of the traps 
occur only once at load/unload time of interceptor, the problem of dangUng intercept (with 
target of IMP InterceptSrv already unloaded) can be avoided since the interroptor can 
afibrd more detailed, time consuming, checks for safe removal of the intercepts (e.g. by 
refusing to unload itself if it is not the last interceptor). Doing these checks in the old 
trapping system is not only time consuming on every API call, but it is in most cases 
extremely difficult, if not inherently impossible, since it would require that the interceptor 
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refuse execution of the original API call, which will cause malfunction in the calling 
application. In addition, the system described herein protects against unauthorized 
canceling of installed API security functions since canceling a new API task installed using 
the technique described herein, without restoring the original opcodes of the original AP^ 
5 will likely cause the system to crash. 



Brief Description Of Drawings 

FIG. 1 is a diagram illustrating a relationship between an old API and a new API 
according to the present invention. 



FIG. 2 is a flow chart showing steps that are performed to install the new API 
10 according to the present invention. 

FIG. 3 is a flow chart illustrating steps that are performed to remove the new API 
that is installed using the steps of FIG. 2. 

Detailed Description of the Preferred Embodiment(s) 

Refer to FIG, 1, a diagram 10 illustrates a relationship between an old API 12 and 

15 a new API 14. The old API 12 represents an existing API provided with an operating 

system, such as MS Windows 95. The new API 14 represents an API that is provided for 
use in connection with, for example, an applications program. As discussed in more detail 
below, the new API 14 can be executed instead of the old API 12 or can be executed in 
addition to the old API 12. Note that, in some instances, the new API 14 and/or the old 

20 API 12 may be referred to herein as a "routine**. However, the term "routine" should not 

be understood as referring to a single, unitary, block of code but, instead, should be 
understood to refer to a collection of code that may be pro\aded in a plurality of blocks 
that may make calls or jumps therebetween. Note also that the specific functionality 
provided by the new API 14 is a design choice but may, in some instances, including 

25 saving and restoring registers used by the caller and/or the old API 12. 
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In order to execute the new API 14, the old API 12 is patched with an 
unconditional jump instruction 16 that transfers control from the old API 12 to the 
be^nning of the new API 14. If the new API 14 is executed instead of the old API 12 
O.e., the old API 12 is not to be executed), then a return to the calling routine occurs at 
5 the end of the new API 14, as indicated by the dotted line shown at the end of the new 

API 14. Note that, as will be apparent to one of ordinary skill in the art, other suitably 
equivalent control flow instructions may be used m place of the unconditional jump 
instruction 16. 

If the old API 12 will be executed iri^dditionto the new API 14, then opcodes * 3 
10 that were located at a relocatable portion of the old API 12 (in this case the beginning of 

the old API 12) become relocated code 18 that is placed at the end of the new API 14. As 
discussed in more detail hereinafter, any relative offsets between opcodes within the 
relocated code 18 and opcodes in the remainder of the old API 12 are adjusted, as 
appropriate. Note that, as will be apparent to one of ordinary skill in the art, it is possible 
15 to place the relocated code 18 at an other portion of memory and then use an appropriate 

control flow instruction at the end of the new API 14 to transfer program control from the 
new API 14 to the relocated code 18. 

Immediately following the end of the relocated code 18 is an unconditional jump 
instruction 19 that transfers program control to the portion of the old API 12 immediately 

20 following the relocatable portion of the old API 12, mariced on the diagram 10 with the 

address "CONT-. Thus, if both the old API 12 and the new API 14 are to be executed, 
then the calling routine calls the old API 12 which jumps, via the jump instruction 16, to 
the beginning of the new API 14 which then executes and, at the end thereof, executes the 
opcodes of the relocated code 18 followed by the jump instrurtion 19 that jumps back to 

25 the remainder of the old API 12. Note that the relocated code 18 and the portion of the 

old API 12 begbning at the CONT address constitute the entirety of the old API 12. 
Also note that, as will be apparent to one of ordinary skill in the art, other suitably 
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equivalent control flow instructions may be used in place of the unconditional jump 
instruction 19. 

Referring to Fig. 2, a flow chart 20 illustrates steps for making patches that cause 
execution of the new API 14 vdien an application program or the operating system calls 
the old API 12, Processing b^ns at a first step 22, v/hcre a byte fi-om the be^nning of 
the old API 12 is fetched. Following the step 22 is a test step 24 \^ch detemiines if a 
^ole instruction (as opposed to a partial instruction) has been fetched. This 
determination is made in a conventional feshion by, for example, disassembling the fetched 
bytes. Note that it is necessary to fetch an integral number of instructions from the 
relocatable portion of the old API 12 since it is not possible to execute a partial 
instruction. 

If it is determined at the test step 24 that one or more whole instructions have not 
been fetched, then control passes from the test step 24 back to the step 22 to fetch another 
b)te. Otherwise, if an integral number of instructions have been fetched, then control 
passes from the test step 24 to a test step 26 which dotemiines if enough bytes have been 
fetched from the relocatable portion of the old API 12 to accommodate the unconditional 
jump instruction 16 that transfers control from the old API 12 to the new API 14, In 
some embodiments, the required number of bytes is five. However, the test step 24 
preceding the test step 26 makes it possible that a number of bytes greater than five will 
have been fetched since it is necessary that a number of bytes corresponding to an integral 
number of instructions be fetched from the relocatable portion of the old API 12. 

If it is determined at the test step 26 that enough bytes have not been fetched, then 
control passes from the test step 26 back to the step 22 where another byte is fetched. 
Otherwise, if enough bytes have been fetched, then control passes from the test step 26 to 
a step 28. Note that it is not possible to reach the test step 26 without having fetched a 
number of bytes corresponding to an integral number of instructions. This is because it is 
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not possible to execute the test step 26 without having passed the test at the step 24, 
A^ch determines that the number of fetched bytes corresponds to a whole number of 
instructions. 

At the step 28, the bytes that have been fetdied are moved from the relocatable 
portion of the old API 12 to the end of the new API 14 and any opcodes in the rdocated 
code 18 that refer to relative offsets are resolved. Note that opcodes in the relocated code 
18 that contain a relative offset, such as a jump relative or a call relative, may need to be 
modified when the opcodes are relocated. Also note that, the relative portions within 
memory of the old API 12 and the new API 14 should not change after the APPs 12, 14 
are loaded in memory. 

Following the step 28 is a step 30 where the unconditional jump instruction 19 is 
added to the end of the new API 14. As discussed above, the unconditional jump 
instruction 19 causes control to return back to the portion of the old API 12 that follows 
the relocatable portion of the old API 12. Followng the step 30 is a step 32 where the 
unconditional jump instruction 16 is added to the bediming of the old API 12 so that 
when an application program or the operating system calls the old API 12, the 
unconditional jump instruction 16 from the old API 12 to the new API 14 will be 
executed. 

As discussed above, it is possible that the new API 14 entirely replaces the old API 
12 so that no part of the old API 12 needs to be executed once the new API 14 has been 
provided. In that case, an alternative patch is provided. As shown in FIG. 2, control 
passes from the test step 26 to a step 34 where a return instruction is added to the end of 
the new API 14 (if a return instruction is not already found at the end thereof). Following 
the step 34, control passes to the step 32 where the unconditional jump instruction 16 is 
added to the relocatable portion of the old API so that a call to the old API 12 \M cause 
program control to flow from the old API 12 to the new API 14. 
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Note that, in some instances, it may be unad\asable to relocate the code that is at 
the beginning of the old API 12. For example, there may be other instructions within the 
old API 12 that reference the code located at the beginning thereof In those cases, it is 
possibletouseotherportions bftheold API 12 as the relocatable portion. For example, it 
5 would be possible to relocate bytes following the first N microprocessor instructions, in 
the manner described above, and replace the relocated bytes with the unconditional jump 
instruction 16. 

The code that executes the patching step illustrated by the flow chart 20 may be 
written in a conventional computer source language, such as C++, and compiled in a 
10 conventional manner similar to compilation of other Nficrosoft Windows DLL's. In some 

instances, the preferred base address of the new API 14 may be set to a value that will 
cause the new API 14 to always load at the same address for all the processes which use 
the new API 14. Thus, the new API 14 may be shared so that the new API 14 is loaded in 
memory only once, even when used by multiple processes. 

15 Refer to FIG. 3, a flow chart 40 illustrates steps that are performed when a process 

that uses the new API 14 is removed from memory. Note that, under the Windows 
environment, a special routine (MS Main) is called when a process is removed from 
memory. The MS Main routine provides the application with an opportunity to do 
cleanup including, in this instance, restoring the old API 12. 

20 Processing begins at a first test step 42 where it is determined if the process being 

removed is the last process that uses the new API 14. If not, then the old API 12 and the 
new API 14 are not modified and no cleanup is done, since the new API 14 must remain 
to be used by the other processes. Otherwise, control passes from the test step 42 to a 
step 44 where the relative instructions of the relocated code 18 are modified back to the 

25 original state prior to restoring the relocated code into the old API 12. Following the step 

44 is a step 46 where the relocated code 18 is restored into the old API 12, thus 
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overriding the unconditional jump instruction 16 that was provided to the old API 12 
\^en the old API 12 Avas patched. Once the relocated code 18 is restored to the old API 
12, then a call to the old API 12 will not result in execution of the new API 14. 

Note that, although the invention has been illustrated herein using APIs with the 
Windows operating system, it would be straight-forward for one of ordinary skiU in the art 
to adapt the system described herein to other operating systems and other types of 
routines. 

While the invention has been disclosed in connection with the preferred 
embodiments shown and described in detail, various modifications and improvements 
thereon will become readily apparent to those skilled in the art. Accordingly, the spirit 
and scope of the present invention is to be limited only by the following claims. 
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1. A method of supplementing a software routine loaded in computer memory, 
comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the software routine to an other memory location, 
the number of bytes corresponding to an integral number of instructions of 
the relocatable portion; 

(c) causing program control to flow from the additional routine to the 
relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

2. A method, according to claim 1, wherein causing program control to flow from the 
additional routine to the relocated opcodes includes placing the relocated opcodes at a 
location in the computer memory that immediately follows the additional routine. 

3. A method, according to claim 1, wherein causing program control to flow from the 
relocated opcodes to the memory address immediately following the relocatable portion 
includes placing a program control instruction at a location in the computer m^ory 
immediately follovwng the relocated opcodes. 

4. A method, according to claim 3, wherein the program control instruction is an 
unconditional jump instruction. 

5. A method, according to claim 1, wherein causing program control to flow from the 
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relocatable portion to the additional routine includes pladng a program control instruction 
at a memory location corresponding to a source address of the relocatable portion of the 
software routine. 

6. A method, according to claim 5, wherein the program control instruction is an 
unconditional jump instruction. 

7. A method, according to claim 6, wherdn providing the relocated opcodes includes 
relocating a number of bytes that is at least equal to an amount of bytes required for the 
unconditional jump instruction. 

8. A method, according to claim 1, further comprising: 

(f) following providing the relocated opcodes, resolving any opcodes 
contained therein that reference relative displacements between the 
relocated opcodes and opcodes contained in the software routine. 

9. A meltiod, according to claim 1, wherdn the software routine and additional routine are 
API's that run under the Microsoft Windows operating system, 

10. A method, according to claim 9, wherein the additional routine is configured to load at 
a predetermined address in the computer memory. 

1 1. A method of supplementing a AVindows API loaded in computer memory, comprising: 

(a) loading an additional routine into the computer memory; 

(b) providing relocated opcodes by relocating a number of bytes ft^om a 
relocatable portion of the API to an other memory location, the number of 
bytes corresponding to an integral number of instructions of the relocatable 
portion; 

(c) causing program control to flow from the additional routine to the 
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relocated opcodes; 

(d) causing program control to flow from the relocated opcodes to a memory 
address immediately following the relocatable portion; and 

(e) causing program control to flow from the relocatable portion to the 
additional routine. 

12. A software program that supplements a Windows API loaded in computer memory, 
comprising: 

an additional routine that is loaded into the computer memory; 

first means for providing relocated opcodes by relocating a number of bytes from a 
relocatable portion of the API to an other memory location, the number of bytes 
corresponding to an integral number of instructions of the relocatable portion; 

second means, coupled to the first means and to the additional routine, for causing 
program control to flow from the additional routine to the relocated opcodes; 

third means, coupled to first means and to the API, for causing program control to 
flow from the relocated opcodes to a memory address inunediately following the 
relocatable portion; and 

fifth means, coupled to the API and to the additional routine, for causing program 
control to flow from the relocatable portion to the additional routine. 
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